Distributed Mean Estimation with Limited Communication
نویسندگان
چکیده
Motivated by the need for distributed learning and optimization algorithms with low communication cost, we study communication efficient algorithms for distributed mean estimation. Unlike previous works, we make no probabilistic assumptions on the data. We first show that for d dimensional data with n clients, a naive stochastic rounding approach yields a mean squared error (MSE) of ⇥(d/n) and uses a constant number of bits per dimension per client. We then extend this naive algorithm in two ways: we show that applying a structured random rotation before quantization reduces the error to O((log d)/n) and a better coding strategy further reduces the error to O(1/n). We also show that the latter coding strategy is optimal up to a constant in the minimax sense i.e., it achieves the best MSE for a given communication cost. We finally demonstrate the practicality of our algorithms by applying them to distributed Lloyd’s algorithm for kmeans and power iteration for PCA.
منابع مشابه
A Robust Distributed Estimation Algorithm under Alpha-Stable Noise Condition
Robust adaptive estimation of unknown parameter has been an important issue in recent years for reliable operation in the distributed networks. The conventional adaptive estimation algorithms that rely on mean square error (MSE) criterion exhibit good performance in the presence of Gaussian noise, but their performance drastically decreases under impulsive noise. In this paper, we propose a rob...
متن کاملImpacts of the Negative-exponential and the K-distribution modeled FSO turbulent links on the theoretical and simulated performance of the distributed diffusion networks
Merging the adaptive networks with the free space optical (FSO) communication technology is a very interesting field of research because by adding the benefits of this technology, the adaptive networks become more efficient, cheap and secure. This is due to the fact that FSO communication uses unregistered visible light bandwidth instead of the overused radio spectrum. However, in spite of all ...
متن کاملDistributed Incremental Least Mean-Square for Parameter Estimation using Heterogeneous Adaptive Networks in Unreliable Measurements
Adaptive networks include a set of nodes with adaptation and learning abilities for modeling various types of self-organized and complex activities encountered in the real world. This paper presents the effect of heterogeneously distributed incremental LMS algorithm with ideal links on the quality of unknown parameter estimation. In heterogeneous adaptive networks, a fraction of the nodes, defi...
متن کاملMean Square Performance of Consensus-Based Distributed Estimation over Regular Geometric Graphs
Abstract. Average-consensus algorithms allow to compute the average of some agents’ data in a distributed way, and they are used as a basic building block in many algorithms for distributed estimation, load balancing, formation and distributed control. Traditional analysis of such algorithms studies, for a given communication graph, the convergence rate (second largest eigenvalue of the transit...
متن کاملInformation fusion strategies and performance bounds in packet-drop networks
In this paper we discuss suboptimal distributed estimation schemes for stable stochastic discrete time linear systems under the assumptions that (i) distributed sensors have computation capabilities, (ii) the communication between the sensors and the estimation center is subject to random packet loss, and (iii) there is no communication between sensors. We consider strategies which are based on...
متن کامل